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This is a full and timely response to the non-final Official Action mailed July 16, 
2003. A request for a one month extension of time and the requisite fee accompany this 
amendment. Reconsideration of the application in light of the above amendments and the 
following remarks is respectfully requested. 

By the forgoing amendment, the specification and claims 7-9 have been amended. 
Additionally, claims 1-6 have been cancelled, and new claims 1 1-25 have been added. Thus, 
claims 7-25 are currently pending for the Examiner's consideration. 

In the outstanding Office Action, the Examiner objected to the specification. This 
problem has been corrected by the present amendment, which includes a substitute 
specification. In preparing the substitute specification, the specification has been carefully 
reviewed for grammatical and idiomatic accuracy, and all needed corrections have been made. 
No new matter has been added. 

Proposed corrections to the drawings have been filed concurrently as required by the 
Examiner. An explanation of the changes made in the drawings is provided above. 

The Examiner rejected claim 7 under 35 U.S. C. § 112, first paragraph as not being 
enabled by the specification as originally filed. Claim 7 was filed in a preliminary 
amendment at the same time the present application as originally filed. Consequently, claim 
7 contains originally- filed subject matter of the present application. To provide additional 
support for claim 7, the language of claim 7 has been added to the specification by the present 
amendment. 

With regard to claim 7, the Office Action contends that a person of ordinary skill in 
the art would not be enabled to use a controller coupled to the image sensor to control a 
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display of two dimensional object scenes corresponding to the image data signals. Applicant 
respectfully disagrees. It seems clear that one of ordinary skill in the imaging art would 
possess knowledge sufficient to couple a controller to an image sensor to control display of a 
two dimensional object scene on the display as recited by independent claim 7. Accordingly, 
Applicant believes independent claim 7 to be clearly enabled one of ordinary skill in the 
imaging art as required by 35 U.S.C. §112, first paragraph. 

The Examiner also rejected claims 7 under 35 U.S.C. §112, second paragraph as 
failing to provide sufficient antecedent basis for the recitation of "a controller coupled to the 
image sensor to control a display of two dimensional object scenes corresponding to said 
image data signals." The Examiner appears to be arguing simply that claim 7 is not supported 
by the specification as filed. As previously stated, claim 7 is an originally-filed claim of the 
present application and the language of claim 7 has been added to paragraphs 10 and 37 of the 
specification. Accordingly, Applicant respectfully requests that this rejection under 35 U.S.C. 
§ 1 12, second paragraph, be reconsidered and withdrawn. 

The Examiner further rejected claims 8-9 under 35 U.S.C. §112, second paragraph. 
No specific language was mentioned with respect to claim 9. These claims have been 
carefully reviewed in light of the Examiner's comments and amended as necessary. 
Following this amendment, all the remaining claims are believed to be in compliance with 35 
U.S.C. § 1 12 and notice to that effect is respectfully requested. 

The Examiner rejected independent claim 10 under the judicially created doctrine of 
obviousness-type double patenting. The Examiner further stated that independent claim 1 0 
would be allowable if a terminal disclaimer in compliance with 37 C.F.R. 1.321 (c) were to 
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be timely filed by applicant. A terminal disclaimer in compliance with 37 C.F.R. 1.321 (c) 
accompanies the amendment. Therefore, allowance of claim 10 is respectfully requested. 

New claims 1 1-25 have been added. Support for claims 11-15 may be found in the 
specification, including paragraphs 1 and 26-35. Support for claims 15-25 maybe found in 
the specification, including paragraphs 1 and 26-35 and in originally-filed claim 10. 

For the foregoing reasons, the present application is thought to be in condition for 
allowance. Accordingly, favorable reconsideration of the application in light of these remarks 
is courteously solicited. If any fees are owed in connection with this paper which have not 
been elsewhere authorized, authorization is hereby given to charge those fees to Deposit 
Account 18-0013 in the name of Rader, Fishman & Grauer PLLC. If the Examiner has any 
comments or suggestions which could place this application in even better form, the 
Examiner is requested to telephone the undersigned attorney at the number listed below. 



Steven L. Nichols, Esq. 
Managing Partner, Utah Office 
Rader Fishman & Grauer PLLC 
River Park Corporate Center One 
10653 S. River Front Parkway, Suite 150 
South Jordan, Utah 84095 

(801) 572-8066 
(801) 572-7666 (fax) 



Respectfully submitted, 



DATE: 14 November 2003 




Steven L. Nichols 
Registration No. 40,326 
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METHOD AND APPARATUS FOR OMNIDIRECTIONAL 
THREE DIMENSIONAL IMAGING 

FIELD OF INVENTION Fi e ld of Inv e ntion 

[0001] This invention presents a set of methods and apparatus for omnidirectional 
stereo imaging. By "omnidirectional imaging system", we mean a system that is able to 
acquire images with a field-of-view covering the entire solid angle of a hemisphere ( 180 s olid 
spac e angl e2ji steradians), simultaneously without any mechanical moving part. The field of 
view of a conventional camera or a light projector can be dramatically increased by 
employing a reflective mirror properly placed in front of the camera or the projector. A pair 
of omnidirectional cameras is able to form a unique stereo imaging system that is able to 
obtain three dimensional images of surrounding scene with 360 degree view angle. A 
combination of an omnidirectional camera and an omnidirectional structured light projector 
can also provide a means to obtain quantitative three dimensional measurements of the 
objects around the camera system. The omnidirectional three dimensional imaging methods 

and apparatus presented herein may offer unique solutions to many practical s^emS! that 

h £ 

need simultaneous 360 degree viewing angle and three dimensional measurement cap ability. 

i ^ rn 

BACKGROUND firie r Art Existing Approaches to Larg e FOV Imagine Syst e m £ 00 — ^ 
[0002] A number of approaches had been proposed in the past fcjr imagigg g> .y 
systems to achieve wide field-of-view (FOV). None of them, however, is ab|e tp ge§£rate 3D 
omnidirectional images. In the following paragraphs, we give a briefly survl^on thestate-of- 
the-art of current imaging systems that seek to achieve wide FOV. 



Conventional Cam e ras 

[0003] Most existing imaging systems employ electronic sensor chips, or still 
photographic film, to record optical image collected by its the imaging systems' optical lens 
system. The image projection for most camera lenses is modeled as a "pin-hole" with a single 
center of projection. Sinc e The sizes of camera lens lenses and the imaging sensor have th e ir 
practical limitations, such that the light rays that can be collected by a camera lens and 
received by the imaging device typically form a eem with cone having a very small opening 
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angle. Therefore, angular field-of-views for conventional cameras are within a range of 5 to 
50 degrees. For example, an 8.5 mm F/1.3 camera lens for 1/2" CCD (Charge Coupled 
Device) chip only has an angular FOV of 41.2 degree. 

Fish Eyo L e ns e s 

[0004] Optical engineers had designed several versions of wide-viewing-angle 
lens system, called the fish-eye lenses Ions (goo [1],[2]) . See Wood. R.W.. Fish-Eve View and 
Vision Underwater, Philosophical Magazine. 12 (Series 6^:159-162. 1906; Miyamoto, K, 
Fish-Eve Lens. J. Optical Soc. America. 54(8^:1060-1061. 1964. Tho fishF ish-eye lens 
lenses f e atur e s feature a very short focal length which, when used in place of conventional 
camera lens, enables the camera to view object for much wider angle (almost 2n 
steradians lSO d e gr ee of hemisph e r e). In general, the a wider FOV- requires a the more 
complicated design for the fish-eye lens has. To obtain a hemispherical FOV, the fish-eye 
lens must be quite large in dimension, complex in optical design, and hence expensive. Also, 
it is very difficult to design a fish-eye lens that ensures single view point constraint, i.e., all 
incoming principal light rays intersect at a single point to form a fixed viewpoint. This is 
inde e d a probl e m with comm e rcial fish e yo lens e s, including Nikon's Fishoy e Nikkor 8 mm 
f/2.8 l e ns. The use of fish-eye lenses for wide FOV imaging application has been advocated 
by [3] and [ 4 ], among oth e rs . See Oh, S.J, and Hall. E.. Guidance of a Mobile Robot Using an 
Omni-directional Vision Navigation System. Proc. SPIE. 852:288-300. Nov.. 1987: U.S. 
Patent No. 5.359.363 issued to Kuban. P.P.. et al. Oct.25. 1994. Although the acquired 
image by fish-eye lenses may prove to be good enough for some visualization applications, 
the distortion compensation issue has not been resolved^-and^ the The high unit-cost issues 
remain to b e as major hurdles for its wide- spread applications. The fish-eye lens technique 
has th e advantag e of adopting adopts a statically positioned camera to acquire a wide angle of 
view. However the nonlinear property resulted resulting from the semi-spherical optical lens 
mapping mak e makes the resolution along the circular boundary of the image very poor? 
whil e. Further, the field of view corresponding to the circular boundary of the image usually 
represents a ground or floor where a high resolution of image is required. 
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Multi Camera System or Rotating Imaging Systems 

[0005] Large field of view of objects may be obtained by using multiple cameras 
in the same system, each points towards pointing in a different direction. However, issues en 
related to seamless integration of multiple images is are further complicated by the fact that 
each image produced by each camera has different centers of projection. The cost for such a 
system is usually high. The image processing required by multiple cameras or by rotating 
cam e ra cameras in order method to obtain a precise information on position and azimuth of 
an object takes a long time, which is not suitable for real-time battle field modeling and 
reconnaissance applications. 

[0006] Another straightforward solution to increasing the FOV is illustrated in 
Fias. 1A-1B. of an An imaging system (100) is to rotat e rotated the entir e imaging system 
about its center of projection (Figure-llO). An image sequence (120) of individual images 
(130) is acquired by the camera imaging systems at different positions are "stitched" together 
to obtain a panoramic view of the scene as seen in Fig. IB . Such an approach has been 
recently proposed by several researchers J>oo [5], [6] [7]). See Chen,S. E.. Quick Time VR- 
An Image Based Approach to Virtual Environment Navigation, Computer Graphics: Proc. Of 
SIGGRAPH 95. 29-38. 1995; McMillam. L. and Bishop, G., Plenoptic Modeling: An Image- 
Based Rendering System. Computer Graphics: Proc. Of SIGGRAPH 95. 38-46, 1995; 
Zheng, J.Y.. and Tsuii.S. Panoramic representation of scene for route understanding. Proc. 10 
Int'l Conf. Pattern Recognition. 1:161-167, 1990. A very interesting approach developed by 
that employs a camera with a non-frontal image detector to scan the world. See Krishnan; 
A. and Ahuia. N.. Panoramic Image Acquisition. Proc. Of IEEE Conf. Computer Vision and 
Pattern Recognition (CVPR-96). 379-384.1996. 

[0007] The first disadvantage of any rotating image system is that it requires the 
use of moving parts, and precision positioning devices. A more serious drawback is that such 
systems lack the capability of simultaneously acquiring images with wide FOV. Although 
such system can acquire precise azimuth information in omnidirectional view, the imaging 
process is time-consuming and the method is not applicable to real-time problems such as 
avoiding collision against moving obstacles or monitoring scene with mobile objects. This 
restricts the use of rotating systems to static and/or non-real-time applications. 

[0008] In contrast, the present apparatus, inv e ntion pr e s e nt e d her e in, called th e 
omnidir e ctional cam e ra, is capable of capturing real-time omnidirectional images without 
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using any moving parts. By "omnidirectional images", we mean images with a field-of-view 
covering entire hemisphere ( 180 solid space 2ji steradians of solid-angle), simultaneously. 
Figur e Fig. 2 A-2B provides a comparison between fields of view of the present 
omnidirectional camera (200) our Omnidirectional Cam e ra , and those of panoramic (210) 
cam e ra and conventional (220) cameras. As one can see, a panoramic camera is still not 
ommidirectional, since it can only provide a wide-angle of FOV at certain time instance^ 
Further, this FOV is not in all directions. On the other hand, the field of view of the 
omnidirectional camera (200) covers an entire hemisphere or 2ji steradians of solid angle. 

Summary of th e Invention SUMMARY 

[0009] The primary objective of present invention is to provide a set of simple 
methods and apparatus to obtain simultaneously omnidirectional stereo images without using 
any moving parts. Accordingly, an improved imaging apparatus for generating a two 
dimensional image, includes a substantially hyperbolic reflective mirror configured to satisfy 
an optical single viewpoint constraint for reflecting a scene, an image sensor responsive to 
the reflective mirror and that generates two dimensional image data signals, and a controller 
coupled to the image sensor to control a display of two dimensional object scenes 
corresponding to the image data signals. The field of view of a conventional camera or a 
light projector can be dramatically increased by employing a reflective mirror properly placed 
in front of the camera or the projector. A pair of omnidirectional cameras is able to form a 
unique stereo imaging system that is able to obtain three dimensional images of surrounding 
scene with 360 degree view angle. A combination of an omnidirectional camera and an 
omnidirectional structured light projector can also provide a means to obtain quantitative 
three dimensional measurements of the objects around the camera system. The 
omnidirectional three dimensional imaging methods and apparatus presented herein may 
offer unique solutions to many practical systems that need simultaneous 360 degree viewing 
angle and three dimensional measurement capability. 

Brief Description of th e Drawing BRIEF DESCRIPTION OF THE DRAWINGS 

[0010] Figur e 1 shows a conv e ntional way to obtain panoramic imag e s: th e 
r.nmpnr . ition of multiple vi e ws of a rotated cam e ra into a panoramic imag e . Fig. 1 A illustrates 
a sequence of multiple views taken by a rotated camera. 
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[0011] Fig. IB illustrates the composition of multiple views of a rotated camera 
into a panoramic image. 

[0012] Figur e 2 gives a comparison of tho Field of View (FOV) among 
conv e ntional cam e ras, panoramic cameras, and the proposed omnidirectional camera. Fig. 2 A 
illustrates the Field of View (FOV) of an omnidirectional camera. 

[0013] Fig. 2B illustrates the FOV of conventional and panaromic cameras. 

[0014] Figure 3 provid e s e xamples of r e fl e ctive convex mirror for Omni 
Directional Imaging. Notic e that th e s e conv e x mirrors do not satisfy tho single viewpoint 
constraint (SVC) condition: Tho (extension of) refl e cted rays do not meet at single viewpoint, 
i.o th e virti^ vi e wpoint vnrinr . with rnyr , ' impinging location on tho mirror. Fig. 3 A 
illustrates a conical mirror for omnidirectional imaging. 

[0015] Fig. 3B illustrates a spherical mirror for omnidirectional imaging. 

[0016] Fig. 3C illustrates a parabolic mirror for omnidirectional imaging. 

[0017] Figur e 1 acquires Omni Directional Imag e from th e OMNI Mirror. A 
vid e o camera plac e d at location C can "soo" objects in an entire hemisphere FOV, from a 
win gin virhinl viewpoint at mirror's focal center O. Fig. 4 illustrates an omnidirectional 
imaging system according to one exemplary embodiment. 

[0018] Figur e 5 shows an embodiment of th e Omnidir e ctional 3D Camera. Fi&_5 
shows an omnidirectional stereo camera according to one exemplary embodiment. 

[0019] Fignr a 6- rimiilnr Variabl e Wav e length Filter (CVWFV Fig. 6 illustrates a 
Circular Variable Wavelength Filter (CVWFV 

[0020] Figure 7: Omni dir e ctional Rainbow Light Proj e ction Using tho OMNI 
Mirror and CVWF Fig. 7 illustrates omnidirectional light projection system according to one 
exemplary embodiment. 

[0021] Figur e 8 : Omnidir e ctional Structured Light 3D Cam e ra Fig. 8 illustrates an 
omnidirectional structured light 3D camera according to one exemplary embodiment. 

Description of tho Pr e f e rr e d Embodim e nt sDET AILED DESCRIPTION 

[0022] To dramatically increas e th e The field of view of an imaging system may 
be increased by , w e propose a som e what unusual approach: using a reflective surface (i. e ., 
such as a convex mirrori- with a properly designed surface profile . Tho field of view of a 
vid e o camera can b e gr e atly increas e d by using r e fl e ctiv e s urface with properly d e sign e d 
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surface shapes. The rear- view mirror in a car is a daily example of using reflective mirror to 
increase the FOV of a driver. 

[0023] There are a number of surface profiles that can be used to produce 
omnidirectional FOV. Figur e 3 lists throe examples: conic mirror, spherical mirror, and 
pnmhnlio mirror. Figs 3A-3C illustrate three examples. Fig. 3A illust rates a conic mirror 
(300-1). Fig. 3B illustrates a spherical mirror (300-2^. and Fig. 3C illustrates a parabolic 
mirror (300-3Y The optical geometry of these convex mirrors provides a simple and effective 
means to convert video camera's planar view into an omnidirectional view around the vertical 
axis of these mirrors, without using any moving part. This is accomplished bv directing light 
ravs (310, 320. 330) from the surrounding to a camera (340). 

[0024] At tho first glance, it appears that th e omnidir o ctional imaging task can b e 
accomplished by using ANY conv e x mirror. Unfortunat e ly, this is not tho cas e . In revi e wing 
some basic of imag e formation, wo know that an An image is a two dimensional pattern of 
brightness (or colors). A satisfactory imaging system mt*st preserv e two preserves essential 
such characteristics as: 

(1) Geometric correspondence: There must be a one-to-one correspondence between 
pixels in an image and point in the scene. 

(2) Single Viewpoint Constraint: Each pixel in the image corresponds to a particular 
viewing direction defined by a ray from that pixel on image plane through a "pinhole" (single 
viewing point). 

[0025] Notic e that, although tho conv e x mirrors list e d in Figure 3 can gr e atly 

increase th e FOV, and may prove ad e quate for c e rtain omnidiroctional scene monitoring 
applications, thoy ar e not satisfactory imaging d e vic e s. Th e so reflecting surfac es do not 
preserve tho single viewpoint con s traint (SVC). For a high quality omnidiroctional imaging 
system, all tho light rays coming in th e omni imager hoad should havo a single (virtual) 
viewing point. 

Design of tho Omni Mirror That Moots tho SVC 

[0026] In this section, wo will discuss a d e sirable convex mirror surface profil e 
that s ati s fi e s th e single vi e wpoint constraint: all tho (ext e nsion s of) light rays r e flected by th e 
mirror must pass through a singl e (virtual) viewpoint. Wo call such a r e flectiv e mirror th e 
OMNI mirror. As discussed, the profiles illustrated Fig. 3 can greatly increase the FOV. and 
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may prove adequate for certain omnidirectional scene monitoring applications. The 
reflecting surfaces illustrated in Figs. 3A-3B do not preserve the single viewp oint constraint 
(SVC). For a high quality omnidirectional imaging system, all the light ravs coming in the 
omni imager head should substantially share a single virtual viewing point. In contrast, the 
light ray extensions (350. 360, 370) do not share a single virtual viewpoint on a rotational 
axis (380) of the system. 

[0027] A single viewpoint constraint shall be broadly understood to mean that all 
the extensions of light ravs reflected by the mirror must pass through a single, albeit virtual 
viewpoint. A reflective surface that meets this constraint and the one-to-one pixel 
correspondence constraint will be referred to as an omni-mirror for ease of reference. 
Accordingly, the omni-mirror (410) is then the solid of revolution obtained by sweeping the 
cross-section about an optical axis (380-1) of the omni-directional imaging assembly (400). 
The function of the omni-mirror (410) is to reflect all viewing ravs (310-1. 320-1, and 330-1) 
to the video camera's viewing center or focal point (430) from the surface of physical objects 
in the field-of-view. This reflection is such that all the incident portions of the light rays 
(310-1. 320-1. and 330-1) that reflect off of the omni-mirror (410) have projections (350-1, 
360-1. and 370-1) that extend toward a single virtual viewing point (440) at the focal center 
of the omni-mirror (410). In other words, the omni-mirror (410) effectively steers the 
viewing ravs (310-1. 320-1, and 330-1) such that the camera (420) eauivalentlv sees the 
objects in the world from a single viewing point (440). 

[0028] L e t us first d e fin e nec e ssary symbols and terminology. As shown in th e 
Figur e 4 , wo us e an off th e shelf video camera with a r e gular l e ns whos e FOV covers entir e 
surfac e of th e OMNI mirror. Since the optical design of cam e ra and lens is rotationally 
symm e tric, all w e need to d e t e rmin e is th e cross section function z(r) that d e fines th e mirror 
s urface cross section profile. Th e mirror is th e n the solid of r e volution obtained by sw ee ping 
the cross section about the optical axis. Th e function of th e omni mirror is to r e fl e ct all 
viewing rays coming from vid e o camera's vi e wing cent e r (focal point, lab e led as C) to th e 
surfac e of physical obj e cts in th e fi e ld of view. Th e k e y featur e of this r e fl e ction is that all 
such r e flected rays must hav e a proj e ction towards a singl e virtual vi e wing point at mirror's 
focal c e nter, labeled as O. In oth e r word s , th e mirror s hould effectiv e ly steer vi e wing rays 
such that the cam e ra o quivalently s ee s the objects in th e world from a single viewpoint Q. 
Fig. 4 illustrates an omni-directional imaging assembly (400) that includes an omni-mirror 
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(410) and an imaging device such as a video camera (420). The video camera (420) may be 
any suitable video camera, such as an off-the-shelf video camera with a regular lens with a 
FOV that covers substantially the entire surface of the omni-mirror (410). Off-the-shelf 
video cameras and lenses are rotationallv symmetric. As a result, the cross-section function 
z(r) that defines the mirror surface cross section profile remains as the undetermined variable. 

[0029J Wo choos e hyp o rboloid as the d e sirable shap e of tho Omni mirrors. A 
w e ll know featur e of a hyp e rbolic curve is that: the ext e nsion of any ray r e flected by th e 
hyp e rbolic curv e originat e d from one of its focal points pa ss es through its anoth e r focal point. 
If w e choos e th e hyperbolic profile for th e OMNI mirror, and place a vid e o camera at its 
focal point C, as s hown in Figur e 4 , th e imaging syst e m will hav e a singl e viewpoint at its 
anoth e r focal point O, as if th o video camera w e r e placed at th e virtual vi e wing location O. 
Fig. 4 illustrates a hyperboloid as shape of the omni-mirror (410). A well-know feature of a 
hyperbolic curve is that the extensions (350-1. 360-1, and 370-1) of any ray (310-1, 320-1, 
and 330-1) reflected by the hyperbolic curve originated from one of its focal points passes 
through its other focal point. If we choose the hyperbolic profile for the omni-mirror such 
that all the extensions (350-1, 360-1, and 370-1) pass through one focal point at the viewing 
point (440), and place the focal point (430) of a video camera (420) at the other focal point of 
the hyperbolic curve, as shown in Figure 4, the imaging assembly (400) will have a single 
virtual viewpoint (440) as if the video camera (420) were placed at the virtual viewing 
location (440). 

[0030] For any _moving parts. The fi e ld of view of a conv e ntional camera or a 
light proj e ctor can be dramatically increas e d by e mploying a r e fl e ctivo mirror prop e rly 
plaeedp oint P in front of the cam e ra or the proj e ctor. A pair of omnidir e ctional cameras is 
able to form a unique ster e o imaging system that is abl e to obtain three dim e nsional imag e s 
of surrounding scene, the image reflected from the omni- mirror (410) to image sensor's image 
plane (470) has the radius of cU: 

d c = Vu 2 + v 2 (2) 
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where u (450) and v (460) are the pixel indexes on the image plane (470). As shown in 
Figure 4, the camera viewing angle 7,. (480) corresponding to the point on the scene is given 
by: 

^-"l (3) 

The incoming light rays z (350-L 360-1, and 370-1) to the camera can be described by a line 
equation: 

z = — / (4) 

tany c 



Omitting details of mathematical derivation based on the OMNI Mirror omni-mirror 
equation (1), we can obtain a simple closed- form relationship between the omnidirectional 
viewing angle a (490) and CCD the camera's viewing angle y c (480) : 



a = tan 



_, 2b c-(b 2 +c 2 )cosy c 
a 2 siny c 



(5) 



[0031] This equation establishes a one-to-one corresponding relationship between 
a and y c . This relationship is important when we perform tri angulation calculation in the 
omnidirectional 3D camera system. 

First Embodiment: Omnidirectional Stereo Camera (OSC) 

[0032] With a pair of th e omnidir e ctional cameras, w e can easily construct an 
omnidir e ctional st e reo camera. Figur eFig. 5 shows a possible configuration of the 
QSGromnidirectional stereo camera (500). Two omni-mirrors (410-1. 410-2) are placed 
face-to-face withand share a common optical axes align e d. Th eaxis (510). Two virtual 
imaging centersrO ^ and 0 37 (440-1. 440-2) are separated by a distance e£(B). which forms 
the baseline for the stereo vision. 

[0033] The triangulation can be carried out directly from omnidirectional images 
without the need for image conversion. Once a 3D object (520) is detected4n-the 
omnidirectional imag e, the viewing angle 71 (480-1) and ¥a-(480-2) can be determined from 
cam e ras' the geometryT of the two cameras (420-1. 420-2). Based on equation (5), the virtual 

9 



80169-0036 



09/884,255 



viewing angles of thio 3D object, ou a, (490-1) and e^^-frem-a, ( 490-2) between the 3D object 
(520) and the virtual viewpoint Q x-and-Q aviewpoints (440-1, 440-2) can be determined. The 
distance between one of the vi o winc ccn teFvirtual viewpoint (440-1, 440-2) and the 3D object 
(520) in the scene cane be calculated using straight forward triangulation principles: , 



cos(or 2 ) 
sinCa, + a 2 ) 



-B 



(6) 



where R is the Range, i.e. distance between Pa point in space, such as one on the 3D object 
(520) and Orsukrthe virtual viewpoints (440- It-, 440-2). 

[0034] Second Embodiment: Omnidirectional Structured Light 3D Camera 
In other implementations (not shown) an improved imaging apparatus for generating a two 
dimensional image, includes a substantially hyperbolic reflective m irror configured to satisfy 
an optical single viewpoint constraint for reflecting a scene, an image sensor responsive to 
the reflective mirror and that generates two dimensional image data signals, and a controller 
coupled to the image sensor to control a display of two dimensional object scenes 
corresponding to said image data signals. 

[ QQ35 ] 4 .5.1 Proj e cting Structured Illumination Using tho OMNI Mirror G ireute 

Variable Wavelength Filter (CVWF) 

[0036] Notice that in our Omni 3D camera Gvotomj ig. 6 illustrates a circular 
variable wavelength filter (CVWF)4s-, 600) which may be used to facilitate the generation of 
360° projection illumination through a convex mirror. The CVWF (600) is a circular optical 
glass plate coated with color rings of gradually varying wavelengths within the visible (400- 
700 nm) or IR (>700nm) region. The wavelengths of the coated color rings are linearly 
proportional to their radkmsradii as measured from the center of the filter glass , plus an 
offset. For example, if the range is to be between 400-700 nm, the wavelength in the center 
of the CVWF would be 400, and would vary linearly with the radius to 700 nm, based on the 
radius of the CVWF. This feature provides a simple and elegant way of generating structured 
light for an entire scene without using any moving parts (Figur e 6). . 

[0037] As seen in Fig. 6, the CVWF (600) may be used with a white light 
projector (610) to produce conical sheets of light (620. 630) of varying wavelengths. The 
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wavelength of the light X passing through a particular position of the CVWF (600) is a linear 
function of r, the radtansradius measured from the center of the filter glass: 

A(r)=A c + ^j^r (7) 

where \ , \ and R are filter parameters: >t is the wavelength corresponding to the filter's 
"center" (lowest wavelength color the filter can generate). \> is the wavelength 
corresponding to the filter's "edge" (highest wavelength color the filter can generate). R is 
the effective radian of the filter's "edge". And r is the radjansradius measured from the center 
of the filter glass. 

{tm$\ If the relative position of the CVWF (600) is fixed with respect to the 

white light projector £610}, the projected light with the same wavelength forms light sheets of 
circular conic shapes. (620. 630V The cone angle #(640) between the light sheet and the 
normal line of the projector (650) is directly related to the wavelength of the light. In other 
words, the wavelength of the light is encoded with the geometric information of the projected 
eemcone angle . This fixed wavelength X to angl e 8 relationship i s tho key idea for our In 
order to obtain a rainbow 3D camera syst e m. 

Omni directional Rainbow Light Projection U s ing the OMNI Mirror 

[0039] In ord e r to obtain a Rainbow like illumination with a 360° omnidirectional 
projecting angle, w e us e an Omni Mirreromni -mirror (410; Fig. 4) may be used to reflect the 
eomeconical -shaped light sheets (620. 630) generated by the white light projector and a 
CVWF. 

[0040] As shown in Figure. 7, the-an omnidirectional rainbow light projection 
system (700) has a 360° effective projecting angle around the vertical axis (710) of the OMNI 
an omni- mirror- (410). 

[0041] The OMNI MiF reromni-mirror mapping provides an easy and elegant way 
to determine the onmni- projecting angle 6 (720) of a-conic light sheetsheets (620-1. 630- 1) 
based on the projection angle -y(730) of the white light projector ( WLPW 610) . The 
projection line equation is given by: 
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r 



-f 



(8) 



z — 



tan y 



Combining with the OMNI Mirror omni-mirror equation yields : 




2 , 

_ = 1, where c = *Ja 2 +b 2 (9) 



a 



the reflecting point ( 740). which is characterized by the coordinates ( Vm P , Z mp ) on the omni- 
mirror (410), can be calculated by solving a quadratic equation. Using some properties of the 
OMNI Mir feFomni-mirror (410) and simplification procedures in mathematical 
manipulations, the relationship between the white light projection angle 7(730), and the 
omni-projection angle 8 (720) can then be expressed as: 



In other words, knowing the white light projection angle y (730) and parameters of the 
OMNI Mire eromni -mirror (410) , the omni-projection angle 6 (720). whic h is taken with 
respect to the virtual projection center (750), is fully determined. Since the value of y £730} 
determines the projected wavelength in the rainbow spectrum \ spatial geometric 
characteristics of the projected cone shape Rainbow rainbow light sheetsheet s (610-U 620-1) 
are fully defined. Although the 3D range calculation of the Omni 3D camera does not 
require the precise relationship between 6 and \ such a concise relationship facilitates a 
simple design, implementation and tests of the omnidirectional rainbow light projector 
(700). 

Omnidirectional Structured Light 3D Camera 

[0042] FigureFig. 8 shows an embodiment of an omni-directional structured light 
3D cam e ra. imaging system (800). An omnidirectional rainbow light projector (700) is used 
her e in to produce a spatially varying wavelength illumination in the surrounding scene. An 
omnidirectional eameraimaging assembly (400) is placed co-axially (i.e., with optical axes 
aligned) with the omni -directional rainbow light projector (700). The virtual projection 



6 = tan 



2bc-{b 2 +c 2 )cos/ 
a 1 sin y 



(10) 
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centersr©4-, (7401, and the virtual imaging center Qa(440) , are separated by a distance ef 
gQT}> which forms the baseline for the triangulation based 3D vision system. The 
triangulation can be carried out directly from the omnidirectional images without the need for 
image conversion. Once a 3D object (520-1) is detected in the omnidirectional image, the 
viewing angle ^(480-3) is determined from cameras' geometry. The projection angle ^(730- 
IXis determined from the wavelength of the structured light projection, due to the one-to-one 
corresponding relationship of the Rainbow omnidirectional rainbow light projector (700) 
between the wavelength and the projection angle. The distance between the virtual imaging 
center Qa(440) and the surface points on the 3D object in the scene can be calculated using 
straight forward triangulation principle: 



sin(aj + a 2 ) 

where Oh corresponds to the omnidirectional viewing angle (490-3) and ob corres ponds to the 
projection angle (720-1). 

[0043] Accordingly, an omnidirectional rainbow light projector and an 
omnidirectional imaging system may be used together to quickly obtain im ages with known 
three-dimensional characteristics about view angle of 360 degrees. The pr eceding description 
has been presented only to illustrate and describe embodiments of invention. It is not 
intended to be exhaustive or to limit the invention to any precise form disclosed. Many 
modifications and variations are possible in light of the above teaching. It is intended that the 
scope of the invention be defined by the following claims. 
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Th o inv e ntion claimed is: WHAT IS CLAIMED IS: 

1. An omnidirectional two dimensional imaging apparatus comprising: 

(a) A truncated convex reflective mirror that reflects an image of substantially 
hemispherical scene; 

(b) An imaging sensor means positioned to receive said omnidirectional images; 
whereby images with wide field-of-view of substantially hemispherical scene from a 

single viewpoint can be obtained. 

2. An apparatus as recited in claim 1 , wherein the reflective mirror is a substantially 
hyperbolic reflective mirror whereby the substantially hemispherical omnidirectional images 
with single viewing center can be obtained. 

3. An omnidirectional stereo camera apparatus comprising of a pair of optically 
aligned omnidirectional two dimensional imaging systems as recited in claim 1 whereby the 
stereo omnidirectional images can be obtained. 

4. An omnidirectional stereo camera apparatus comprising of a pair of optically 
aligned omnidirectional two dimensional imaging systems as recited in claim 2 whereby the 
stereo omnidirectional images can be obtained. 

5. An omnidirectional three dimensional camera apparatus comprising: 

(a) An omnidirectional two dimensional imaging systems as recited in claim 1 ; 

(b) An omnidirectional structured light projection means; 

whereby the three dimensional measurement of the surrounding objects in the 
omnidirectional scene can be obtained. 

6. An omnidirectional three dimensional camera apparatus comprising: 

(c) An omnidirectional two dimensional imaging systems as recited in claim 2; 

(d) An omnidirectional structured light projection means; 
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whereby the three dimensional measurement of the surrounding objects in the 
omnidirectional scene can be obtained. 

7. An improved imaging apparatus for generating a two dimensional image, 
comprising: 

a substantially hyperbolic reflective mirror configured to satisfy an optical single 
viewpoint constraint for reflecting a scene; 

an image sensor responsive to said reflective mirror and that generates two 
dimensional image data signals; and a controller coupled to thesaid image sensor to control a 
display of two dimensional object scenes corresponding to said image data signals. 

8. The improved imaging apparatus of claim 7, wherein said hemispherical 
image data signals generated by said sensor are projected from a single virtual viewingpoint 
at the focal center of said hyperbolic mirror. 

9. The improved imaging apparatus of claim 7, wherein said substantially hyperbolic 
reflective mirror is a substantially convex mirror and wherein said image data signals 
generated by said image sensor m e ans are projected from a single virtual viewing point at the 
focal center of said convex mirror. 

10. An omnidirectional stereo imaging system,comprising: 

a first camera that generates hemispherical image data signals; 
a first substantially hyperbolic reflective mirror optically associated with said first 
signal generator such that said first camera views objects in an entire hemispherical field of 
view from a single virtual viewpoint at the focal center of said first reflective mirror; 

a second camera that generates a second set of hemispherical image data signals; 
a second substantially hyperbolic reflective mirror optically associated with said 
second camera such that said camera views objects in an entire hemispherical field of view 
from a single virtual viewpoint at the focal center of said second reflective mirror; and 

a data generator responsive to said hemispherical image data signals from said 
first and second camera for generating three-dimensional data for objects in said 
hemispherical fields of view of said first and second reflective mirror. 
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1 1 . The system of claim 10. wherein said first and second cameras point in opposite 
directions. 

12. The system of claim 10, wherein image data signals correspond to acquiring a 
field of view simultaneously covering 360 degrees of viewing angle. 

13. The system of claim 10, wherein a focal center of said first camera and said first 
hyperbolic reflective mirror are at focal points of a parabolic curve. 

14. The system of claim 13, wherein a focal center of said second camera and said 
second hyperbolic reflective mirror are at focal points of a second parabolic curve. 

15. The system of claim 14, wherein said parabolic curve and said second parabolic 
curve are substantially similar. 

16. An omnidirectional two dimensional imaging system, comprising: 

a reflective surface configured to reflect light rays such that extensions of said light 
rays are substantially coincident on a single viewing point; and 

an imaging system configured to cover the entire surface of said omni-mirror. 

17. The system of claim 16, wherein said reflective surface comprises a substantially 
hyperbolic mirror. 

18. The system of claim 17, wherein said hyperbolic mirror first and second focal 
points in which said single viewing point is at said first focal point and a focal point of said 
imaging system is at said second focal point. 

19. The system of claim 17, wherein said imaging system comprises a camera. 

20. The system of claim 16, wherein said imaging system is configured to capture an 
image through a 360 degree viewing angle. 
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21. An omnidirectional stereo imaging system, comprising: 

a first omnidirectional imaging assembly having a first imaging device and a first 
omni-mirror wherein a viewing point of said omni-mirror and a focal point of said imaging 
device are disposed on focal points of a hyperbolic curve; 

a second omnidirectional imaging assembly having a second imaging device and a 
second omni-mirror wherein a viewing point of said omni-mirror and a focal point of said 
imaging device are disposed on focal points of a hyperbolic curve 

wherein focal centers of said first and second omni-mirrors and focal points of said 
first and second imaging devices are substantially coaxial; and 

wherein said virtual viewing points are separated by a predetermined distance. 

22. The system of claim 21, wherein said imaging systems comprise first and 
second cameras that generate hemispherical image data signals. 

23. The system of claim 22, further comprising a data generator responsive to 
said hemispherical image data signals from said first and second imaging devices for 
generating three-dimensional data for objects in fields of view of said first and second omni- 
mirrors. 

24. The system of claim 22, wherein said first omni-mirror comprises a 
substantially hyperbolic reflective mirror optically associated with said first imaging device 
such that said first imaging views objects through a 360 degree field of view from a first 
virtual viewpoint at a focal center of said first camera and said second omni-mirror comprises 
a second substantially hyperbolic reflective mirror optically associated with said second 
imaging device such that said second imaging device views objects in through a 360 degree 
field of view from a second virtual viewpoint at the focal center of said second reflective 
mirror. 

25. The system of claim 21, wherein said first and second imaging devices 
comprise cameras. 
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ABSTRACT 

An omnidirectional stereo imaging system includes a first omnidirectional imaging 
assembly having a first imaging device and a first omni-mirror, a second omnidirectional 
imaging assembly having a second imaging device and a second omni -mirro r wherein focal 
centers of the first and second omni-mirrors and focal points of the first and second imaging 
devices are substantially coaxial and the virtual viewing points are separated by a 
predetermined distance. 
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